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In November 1996, the National Association of Elementary School Principals approved the 
revision of the Administrative Diagnostic Inventory (ADI), the tide given to the diagnostic 
procedure conducted within their assessment center. To assist the NAESP in the use of the ADI 
for the assessment of current and prospective school principals, this study verifies the construct 
validity of the instrument. 



A CONSTRUCT VALIDATION OF THE 
ADMINISTRATIVE DIAGNOSTIC INVENTORY 



INTRODUCTION 

The purpose of the study was to provide construct validation of the revised Administrative Diagnostic 
Inventory (ADI), since no validation studies existed. The validation was necessary to assure scores achieved on 
individual items (descriptors) correlate favorably with one another on the 13 constructs and that the newly designed 
constructs maintain individual identity. The ultimate goal of the study was to confirm the discreteness of the 13 
behavioral constructs first identified and the intercorrelation of the eight descriptors identified within each construct. 
In other words, the study sought to determine whether the 13 constructs measure separate and distinct characteristics 
of effective school principals and whether the eight descriptors within each construct are intercorrelated and measure 
one specific construct. 

Data for the studv originated from a paper pencil instrument administered to teachers and entry level 
administrators who evaluated their supervising principal. The documentation of construct validity resulted from 
subjecting the survey data to confirmatory factor analysis, using R factor analysis with Kaiser normalization and 
Obiimin rotation. 

With the commitment of the National Association of Elementary School Principals (NAESP) to use the 
ADI for the assessment of current and prospective principals, construct validity was crucial and essential. The study 
not only verifies the construct validity of the instrument, but enhances the likelihood of a more effective and 
consistent measurement of the performance of elementary school administrators. The results of the study will assist 
the NAESP and assessment centers across the country with the goal of using valid measures to assess educational 
administrators. 

THEORY/RATIONALE 

Research indicates that the school principal is recognized as a significant factor in school success. The 
quality of teaching and student learning, the school climate, and the confidence in public schools are direcdy related 
to the performance of school principals (Lipman, 1981; Smith & Andrews, 1989). The importance of school 
principals necessitates a need for effective ways to evaluate and assess school administrators. The field s 
conceptualization of organizational processes, including the leadership construct, is constantly evolving (Gasman & 
Heck, 1992; Hallinger, 1992; Leithwood & Hallinger, 1993). The assessment of leadership and management skills 



and abilities is now recognized as a potential tool in the preparation, development, selection, and professional 
growth of school principals (Edmonds, 1982). 

One method of identifying and measuring administrative potential is the assessment center, a model created 
with simulations and exercises to assess skills and behaviors of individuals (Elsaesser, 1990). The field of 
education developed performance assessment systems to address the heightened concerns for schooling and for 
finding good administrators (Sirotnik & Durden, 1996). The creation of the ADI was in response to the demands 
for principal effectiveness substantiated by stringent accountability- measures. 

Showing the relationship between variables and constructs represents a very important part of the content of 
any scientific field (Comrey, 1973). In some fields, such as education, less agreement exists among researchers 
concerning what variables relate to each other. The relationships between variables are often less clearly defined. 
Factor analysis represents a growing body of statistical methods that can be of great value in these areas (Hair, 

1979). Factor analytic methods can help educational researchers define their variables and relate them to each other 
in an attempt to develop their science to a higher level. 

A research and writing team of eight professional educators addressed the content validity of the revised 
ADI in late 1995 and early 1996 (Coleman & Creighton, 1996). By verifying the construct validity of the 
instrument, districts will gain in planning relevant and productive staff development for school principals. In 
addition, educational leadership institutions can improve the development of administrative training programs. 

JThe purpose of the study was to use confirmatory factoring procedures to conduct a construct validation of 
the ADI. The research questions derived from the rationale and used to focus the study were: 

1. Are the eight descriptors within each of the 13 constructs intercorrelated and truly measuring 

the specific construct? 

2. Are the 13 constructs on the ADI discrete and separate measures of effective elementary school 

principals? 

NEED FOR ADI REVISION 

Validity studies of the ADI (Dale, 1988; Menrad, 1990; Elsaesser, 1990; James, 1991; Durden, 1994; 
Coleman & Petrulis, 1994; Sirotnik & Durden, 1996) emphasized a need for construct validation. Though research 
indicated limited data on various elements of the ADI in dissertations, no construct validation studies were 
completed (Coleman, 1995). 
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California State University Fresno was the first franchise signed by the National Association of Elementary 

School Principals (NAESP) to use the ADI after the rights had been purchased from the University of Washington. 

By October 1995, CSUF had conducted 13 assessments covering approximately 150 candidates (Coleman, 1995). 

From the beginning, the validity and reliability of the ADI was pursued. Though the ADI was believed to 

have been created around sound educational theory, and by a wide variety of educational experts and practitioners, 

research data initiated concerns regarding construct validity. 

In November 1995, Donald Coleman prepared a report to the NAESP detailing three construct validation 

studies. The first two showed similar results: Constructs were very difficult to decipher. 

After considerable delay, we finally have completed the factor analysis on the ADI. We had a great deal of 
difficulty getting the factor analysis program to run because the data continually failed to reach convergence. 
The difficulty appears to have resulted from too many very small independent constructs found within the 
data, (letter from Don Coleman to Ron Aregaldo, Associate Director of the National Association of 
Elementary School Principals, 1994) 

The third study (Coleman, 1994) subjected each of the 12 constructs to individual construct validity 
measures with equally disappointing results. Thirty-seven constructs were found instead of the purported 12 on the 
ADI instrument. Descriptors did not load on appropriate constructs and little variance was accounted for by the first 
six factors (3 1.9%). The report also emphasized the need for revision of the ADI and the inclusion of accompanying 
validity studies of the instrument before presenting it to the NAESP for assessment use. 

A major concern pertained to the fact that the definition stems of the constructs appeared to suggest 
multiple rather than singular measures. Each of the constructs displayed a variety of phrases supposedly describing 
one specific construct. This resulted in the construct measuring several rather than a single variable (Coleman, 

1995). If multiple stems are used, as in the case of the ADI, the data may sort into multiple factors on a single 
construct as indicated by so many factors showing up in the factor analysis. 

The report indicated that the content validity of the ADI was not a problem, but upon closer investigation, 
found that the constructs did not measure the whole range of principal effectiveness. Some of the constructs were not 
significant to principal effectiveness and others appeared to overlap each other. Both of the factor analyses run by 
Petrulis and Coleman in 1994 indicated not only problems with the constructs, but the descriptors also failed to 
adequately measure the constructs. 

In order to further validate each ADI construct, a series of 12 factor analyses were conducted in the fall of 
1995 at CSUF. Each individual construct was factor analyzed separately to determine how well the descriptors 
created to measure the construct loaded on each of the 12 constructs. The reason for implementing this procedure 
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was because the initial factor analyses produced such disappointing results. The purpose was to determine how well 
the descriptors measured the specific construct. The set of factor analyses resulted in the following findings: 

1. Multiple stems (definitions) were found in each construct. This caused the data to sort into multiple 
factors on a single construct which explains why so many factors showed up in the analysis. The weakness 
of the factor analysis appears directly related to the multiple constructs found in the construct definition. 

2. Individual descriptors appeared to overlap across constructs, thus creating weak correlations between 
discriptors written for specific constructs. 

In discussing the former ADI, Sirotnik and Durden (1996) point to the issue of construct validity and 

suggest that the 12 ADI dimensions do not appear to be independent measures of leadership as presently defined and 

operationalized. They contend that the faulty dimensions may be improved by including more involvement from 

other educator groups (such as teachers and school-based administrators) in the refinement and development of the 

ADI. Durden's recent research on the ADI (1994) emphasizes the importance of regular and constant evaluation and 

revision of assessment center models of performance evaluation: 

Leadership expectations are changing. Analysis of assessment centers should be regular and updated 
periodically, every four years or so. This is another reason for further research of the ADI and NASSP 
systems, since the original job analysis was conducted more than five years ago. (p. 252) 



FACTOR ANALYSIS 

The use of factoral analysis to provide construct validation is not new. In Educational and Psychological 
Measurement, Guilford (1946) stated; 

The factorial validity of a test is given by its loadings in meaningful, common, reference factors. This is 
the kind of validity that is really meant when the question is asked “Does this test measure what it is 
supposed to measure?” (pp. 437-438) 

Factor analysis is used in this study to check for discreteness among constructs and intercorrelations among 
items being measured. The validation procedure provides additional assurance that what is being measured is valid. 
Validity is critical in research and research instrument construction. Researchers desire to know whether instruments 
are valid because the instrument data substitute for and act as reality. 

Factor analysis has two primary functions in data analysis (Nunnally, 1978). One function is to identify 
underlying constructs in the data. As an example, the analysis may show the variables of “seeks clarity in purpose” 
and “involves stakeholders in the planning process” to actually be indicators of the same theoretical construct (e.g., 
planning). A second role of factor analysis is simply to reduce the number of variables to a more manageable set. 

In reducing the number of variables, factor analysis procedures attempt to retain as much information as 
possible and to make the remaining variables as meaningful and as easy to work with as possible. The general 
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purpose of factor analytical techniques is to find a way of condensing (summarizing) the information contained in a 
number of original variables into a smaller set of composite factors with a minimum loss of information. 

Confirmatory Factor Analysis 

There are two basic models of factor analysis, both of which are used in measurement and validity studies: 
exploratory factor analysis and confirmatory factor analysis. Exploratory validation occurs when multiple items are 
created for unknown constructs. The data are then analyzed to determine what constructs, if any, can be identified 
among the measures. In confirmatory validation, the constructs are previously identified and have items created to 
specifically measure the constructs. The factor analysis procedure in confirmatory validation statisically analyzes 
both the constructs for discreteness and the items for correlation (Nunnally, 1978). 

METHODOLOGY 

In November 1995, the National Association of Elementary School Principals (NAESP) accepted a 
proposal by Dr. Donald Coleman to revise the ADI assessment instrument. Education administration professors 
from five universities, practicing school superintendents and principals, research consultants, assessment center 
directors, and representatives from NAESP began the process in November 1995, and completed the new ADI 
instrument in February 1996 (Creighton, 1996). 

Step 1 

The initial assignment of the writing and research team was to identify constructs believed important to 
school administrators. The team began by researching constructs and descriptors used by other organizations, 
various national accreditation agencies, task forces, and other assessment instruments. Previous ADI assessment 
instruments, National Association of Seconday Principals (NASSP) assessment dimensions, National Council for 
the Accreditation of Teacher Education (NCATE) standards, National Policy Board for Educational Administration 
(NPBEA) competencies, and a literature review were studied. The purpose of examining various recommended 
competencies was to identify and select the most appropriate and applicable constructs to be used for the ADI. 

The writing team analyzed the existing models of proficiencies, standards, domains, and dimensions to 
form appropriate constructs for the ADI. The team first examined existing constructs individually and then 
collectively to determine those believed to be most important. The result was an agreement of 13 constructs 
(dimensions), or knowledge and skills of effective school leaders. These skills covered both management (personal 
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skills) and leadership (inter-personal) skills. The constructs are listed in Figure 1, along with the 12 original ADI 
constructs for comparison. 

Step 2 

The next step was to define the newly identified constructs. The writing team studied previously written 
definitions from the competency and standards documents. The team maintained that many of the original 
definitions were too broad and contained several stems which would cause overlap with other constructs and 
definitions, resulting in weak validity and reliability. The creation of single stem definitions resulted in a more 
clear and concise definition. An example is provided in Figure 2, illustrating two definitions for the construct 
“creativity,” and also identifying multiple vs single stem definitions. Previous research (Andrews, 1965; Brubaker, 
1983 . Coleman, 1995) indicates that content validity is decreased with such multiple stem constructs. 



REVISIED ADI 


PREVIOUS ADI | 


1. Planning 


Problem Analysis 


2. Organizing 


Judgement 


3. Problem Solving 


Organization Ability 


4. Decisiveness 


Decisiveness 


5. Creativity 


Group Leadership 


6. Systems Analysis 


Instructional Leadership \ 


7. Vision 


Creativity 


8. Communications 


Oral Communications 


9. Instructional Leadership 


Written Communication 


10. Group Leadership 


Human Relations 


1 1. Climate Development 


Resourcefulness 


12. Moral Responsibility 


Educational Values j 


13. Instructiuonal Analysis 


1 



FIGURE 1 

ADI Construct Comparison 



CREATIVITY (original ADI) 

Generates and recogizes innovative solution in work-related situations; exhibits openness 
to new ideas from others; demonstrates originality in developing policies and procedures; 
performs under pressure and during opposition; thinks well during oral interchange; 
maintains effectiveness in varying environments with various tasks, responsibilities, 
and people; exhibits behavior flexibility; adjusts time schedule as need arises; and has 
tolerance for ambiguity. 



CREATIVITY (revised ADI) 

Demonstrates innovation and inventiveness in work-related situations. 



FIGURE 2 

Multiple vs Single Stem Definitions 



Step 3 

After reaching agreement on 13 constructs, the team members individually wrote “descriptors” which 
would provide differing measures for each construct. When finished, the group reviewed the list for selecting the 
most relevant. The task was complex and required a great deal of rework on the items. As an example, the team 
agreed upon a descriptor for the construct of creativity: open and receptive to nontraditional sources of information. 
After considerable debate and discussion, a team member asked how the descriptor of “open and receptive could be 
observed in principal behavior? The team agreed with the concern and rewrote the descriptor to read: demonstrates 
openness and receptivity to nontraditional sources of information.. The descriptor of “open and receptive was not 
specific enough for observation. The team agreed that under certain circumstances, the act of demonstrating 
openness and receptivity could be observed in the behavior of a principal. The issue for the team was to select 
descriptors which were specific, interrelated with each other, and measured the construct. 

Step 4 

In an attempt to improve the inner-item correlations and construct validity of the constructs, the team used 
truth tables to verify that each descriptor would logically agree with others measuring the same construct. A truth 
table resembles a correlation matrix but allows individual items to be compared according to verbal construction 
(logic) by subjecting them to “if/then” statements (Coleman, 1995). 

When using the truth tables to compare dimensions or constructs, two-way disagreements are desired since 
constructs should be discrete and measure different behaviors. Figure 3 displays the creation of *if/then statements 





between the constructs of instructional leadership and creativity. If the two constructs are truly discrete and separate 
measures of effective school leadership, the “if/then” statements should result in a disagreement. The disagreement 
should be a two-way disagreement, which means the reversed “if/then” statement should also result in a “no, or not 
necessarily.” A principal who works effectively with the school community does not necessarily demonstrate 
innovation and inventiveness. Reversing the statement also results in a negative response. The truth table in this 
situation can be used to identify discreteness and help assure differences between constructs. 



INSTRUCTIONAL LEADERSHIP 

works effectively with the school community to advance student learning. 

CREATIVITY 

demonstrates innovation and inventiveness in work-related situations. 

IT then statements: 

1. If a principal works effectively with the school community to advance student learning, then he/she 
demonstrates innovation and inventiveness in work-related situations. 

2. If a principal demonstrates innovation and inventiveness in work-related situations, then he/she w'orks 
effectively with the school community to advance student learning. 

FIGURE 3 

Creating a Truth Table with Two ADI Constructs 

Table 1 illustrates a truth table with the construct of instructional analysis and the eight matching 
descriptors. When using truth tables to strengthen correlation between descriptors, two-way agreements are desired. 
A two-way agreement is desired to best predict high inter-item correlation. The intent of the truth table process was 
further refinement and tightening of constructs and descriptors of the Administrative Diagnostic Inventory. 

Pointing out a distinction between two-way agreements and two-way disagreements may help clartify the 
appropriate use of truth tables. In using truth tables to improve the intercorrelation of items, as in the case of the 
ADI decsriptors, two-way agreements are desired. Two-way agreement across the entire matrix measuring a single 
construct should reflect reasonably high inter-item correlation. However, when the desire is to maintain 
discreteness, as in the case of the ADI constructs, two-way disagreements are necessary. To retain discrete 
constructs in a multi-construct design, care must be maintained to create constructs that are independent of each 
other and possessing items reflecting low levels of statistical correlation with items from other constructs (Coleman. 
1996). Truth tables should reflect two-way disagreement among the constructs when low level of inter-item 



correlation is desired. 



TABLE 1 

Completed Truth Table with ADI Construct and Descriptors 



INSTRUCTIONAL ANALYSIS: works effectively with teachers to improve instruction. 



Truth Table 


1 


2 


3 


4 


5 


6 


7 


8 


1. encourages instructional 
improvement 


XX 


A 


A 




A 


A 


A 


A 


2. provides suggestions helpful to 
teachers for improving instruction 


A 


XX 


A 


R 


A 


.R 


A 


A 


3. possesses high expectation for 
student improvement 


A 




XX 


A 


A 


A 


A 


A 


4. demonstrates interest in student 
intellectual growth and development 




.R 




XX 


A 




A 


A 


5. emphasizes student achievement 


A 




A 


A 


XX 


A 


A 


A. 


6. demonstrates expertise in analyzing 
teaching and learning 


A 


.R 


A 


A 


A 


XX 


A 


A 


7. maintains expertise of alternative 
instructional practices 


A 




A 


A 


A 


A 


XX 


A 


8. possesses knowledge of activities 
occurring in various classrooms 


A 




A 


A 


A 


A 


A 


XX 



Note. 



The numbers across the top of the table represent the eight descriptors, and the “XX” are placed in the box 
where the descriptor correlates with itself. Obviously, the descriptor correlated with itself would result in a 
1.00 correlation. The purpose of this process is to verbally correlate the descriptors with each other and the 
construct. 

Starting with a correlation between descriptor 1 and descriptor 2 (second box across the first row), an “iffthen” 
statement is created to read, “if a principal encourages instructional improvement (descriptor 1), then he/she 
provides suggestions helpful to teachers for improving instruction (descriptor 2).” If the statement is agreed to, 
an “A” is placed in the box representing agreement. If the statement is not agreed to, a “dot” is placed in the 
box representing one-way disagreement. The evaluator moves on to the third box across the first row, until 
each item serves as the consequence to the original item serving as the antecedent. 

As the evaluator begins to address the columns, a need exists to reverse the “iffthen” statement. For instance, 

1 across correlated with 2 down is the reverse “if/then” statement as 2 across correlated with 1 down. If a 
two-way disagreement is encountered, an “R” (rejection) is placed in the cell. A “dot” indicates one-way 
disagreement; an “R” indicates two-way disagreement; and an “A” represents two-way agreement. 

A two-way agreement has the best chance of having high inter-item correlation. A one-way agreement drops 
the probability of a high correlation coefficient while two-way rejection drops the coefficient even further. 
Descriptors which have two-way disagreements recorded (R), are considered poor and either reworked 
or discarded. 



DATA COLLECTION: SURVEY/QUESTIONNAIRE 



A paper pencil survey instrument using the 104 ADI descriptors assisted in the collection of data on 
practicing school administrators and the verification of construct validity of the ADI. The intent was to further 
purify the descriptors and constructs and collect data to be used in the actual construct validity study of the ADI. 
Methodically placing the 104 descriptors throughout the survey prevented the respondents from recognizing any of 
the 13 individual constructs. The intent was to allow the respondent to answer the item honestly and without bias 
or preconceived notion. Survey respondents rated their current administrator with the 104 descriptors on a 6-point 
Likert scale. Testing the relationships between descriptors and the construct was one of the purposes of the survey. 
Responses showing the ratings of eight descriptors to be similar strengthened the construct validity of the 
instrument. 

Two-hundred-three teachers enrolled in the educational administration programs at California State 
University Fresno, and teachers and vice-principals in the California School Leadership Academy completed the 
survey during February and March 1996. Formatting the responses on diskette with Microsoft Excel 4.0 allowed for 
the transfer of data to the Statistical Package for the Social Sciences (SPSS v6. 1). The study subjected the data to 
R factor analysis with Kaiser Normalization and Oblimin rotation. By subjecting the data from the 203 survey 
respondents to factor analysis, the intent was to confirm the correlations between individual constructs and 
descriptors, and address the two research questions developed for study direction and focus. 

The use of factor analysis had two functions in the study. The first was to identify underlying constructs in 
the data. A second role was to simply reduce the number of variables to a more manageable set. In reducing the 
number of variables, factor analysis procedures attempt to retain as much of the information as possible and to make 
the remaining variables as meaningful and as easy to work with as possible. The factor rotation scheme selected in 
the study was Oblimin Oblique to maximize the interpretation of factor loadings (Adams, 1996). 



FINDINGS AND INTERPRETATIONS 

Addressing the first research question involved running an individual correlation matrix on each of the 13 
constructs with the accompanying eight descriptors within each construct. Analyzing the resulting data determined 
the extent of correlation between descriptors and the degree to which the descriptors measured the construct. The 
data also revealed whether the eight descriptors represented one single factor. Two or more factors represented 
within the eight descriptors indicated a construct as too broad and complex. 
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Analyzing the correlation matrices of the 13 ADI constructs resulted in strong intercorrelation of items 
within 11 constructs. The correlation coefficients were well above the required significance level of .18, with most 
ranging from .60 to .80 indicating a high degree of correlation among descriptors. As an example, the ADI 
construct of creativity is shown in Table 2. The data reveal all coefficients above .18 and with only one extracted 
factor possessing an eigenvalue of 5.47. Analyzing each variable with the factor indicates a high loading on the 
factor, ranging from .73 to .91. The data suggest the items as well written and effective measures of the specific 
construct, and identified a single factor as predicted by the research design (Creighton, 1996). In addition, the eight 
descriptors within 1 1 constructs loaded on one specific factor (ranging from .70 to .90), further suggesting the items 
as clear measures of one skill dimension. 



TABLE 2 

Correlation Matrix - ADI Construct of CREATIVITY 







V004 


V017 


VO30 


V043 


V056 


V068 


V081 


V095 


VO04 


1.00 




















VO 17 


.77 




1.00 
















VO30 


.73 




.69 


1.00 














V043 


.66 




.72 


.68 


1.00 












V056 


.77 




.76 


.73 


.71 


1.00 










V068 


.63 




.67 


.55 


.59 


.77 


1.00 








V081 


.61 




.50 


.63 


.53 


.63 


.52 




1.00 




V095 


.50 




.47 


.47 


.64 


.60 


.57 




.66 


1.00 


Variable Communality 


Factor 


Eigenvalue 


Percent of Variance 




Variable 


Factor 1 


VO04 .75 




1 




5.47 


68.3 








VO04 


.86 


V017 .85 




2 




.73 


9.1 








V017 


.85 


VO30 .83 




3 




.53 


6.7 








VO30 


.83 


V043 . 84 




4 




.44 


5.5 








V043 


.84 


V056 . 91 




5 




.27 


3.4 








V056 


.91 


V068 .80 




6 




.21 


2.7 








V068 


.80 


V081 .76 




7 




.18 


2.3 








V081 


.76 


V095 . 73 




8 




.16 


2.1 








V095 


.73 



The communality tables further support the strength of the descriptors written for the 1 1 constructs. The 
high communality coefficients indicate that the descriptors have a large amount of variance accounted for by the 
factor. In other words, the descriptors have much in common with the other descriptors included in the analysis. 



In summarizing the previous data, a restatement of the first research question may be helpful: Are the eight 
descriptors within each of the 13 constructs intercorrelated and measuring the specified construct? The correlation 
matrices of the ADI constructs revealed the descriptors of 1 1 constructs revealed correlation coefficients ranging from 
.47 to .96 and measuring the constructs as written. These 11 constructs also loaded on one separate factor While 
several of the constructs have some items with lower correlation coefficients on the correlation matrix, the coefficients 
exceed the . 18 level of significance. 

The two constructs of climate development and instructional analysis however, resulted in two factors 
identified with eigenvalues greater than 1.00. The rotated factor matrix of these two factors resulted in some 
variables loading highly on one factor and other variables loading highly on a second factor. The factor analysis 
computer program does not identify names or labels for these factors, but points to the ambiguity within the eight 
descriptors. The implication is that the eight descriptors are not measuring the same construct. In other words, the 
descriptors are branching into two separate areas, rather than all measuring one discrete construct. 

Unrotated Factor Analysis 

The first step in addressing the second research question invlolved running an unrotated factor analysis, or 
principal component analysis, of the 104 descriptors (variables) to identify factors represented in the data. The result 
was the identification of 17 factors with eigenvalues greater than 1.0, four more than the desired 13. The second step 
subjected the 17 factors to an oblique rotation to improve the interpretation of data. The two factor matrices 
resulting from this method were (a) a pattern matrix and (b) a structure matrix, which display the loadings of each 
individual descriptor. Pattern loadings best show what variables are highly involved in what clusters and the 
structure loadings measure the correlation of the variables with the factors. The structure matrix gives a better 
definition of oblique patterns (Thompson, 1996). 

In factor analysis, only the factors having eigenvalues of 1.0 or greater are considered significant. All factors 
with eigenvalues less than 1 .0 are considered insignificant and disregarded. The rationale for the eigenvalue criteria 
is that any individual factor should account for at least the variance of a single variable if it is to be retained for 
interpretation. Table 3 illustrates the unrotated factor matrix of the 104 descriptors (variables) which identifies 17 
factors with eigenvalues and percent of variance of 1.0 or greater. Factors 18 through 104 are considered insignificant 
due to their eigenvalues and percent of variance being less than 1.0, and are disregarded for interpretation. 
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TABLE 3 



Factor 


Eigenvalue 


Pet. of Variance 


Cum. Pet 


1 


46.17 


44.4 


44.4 


2 


5.07 


4.9 


49.3 


o 

J 


4.41 


4.2 


53.5 


4 


3.54 


3.4 


59.9 


5 


3.36 


3.2 


60.1 


6 


3.05 


2.9 


63.1 


7 


2.92 


2.8 


65.9 


8 


2.28 


2.2 


68.1 


9 


2.18 


2.1 


70.2 


10 


2.14 


2.1 


72.2 


11 


1.83 


1.8 


74.0 


12 


1.56 


1.5 


75.5 


13 


1.38 


1.3 


76.8 


14 


1.31 


1.3 ‘ 


78.1 


15 


1.19 


1.1 


79.2 


16 


1.12 


1.1 


80.3 


17 


1.01 


1.0 


81.3 



The basic ‘'unrotated” factor analysis employs principal components analysis (also termed principal factor 
analysis). The objectives of the principal components is to generate a first factor that has the maximum explained 
variance. With the first factor and its associated loadings fixed, principal components locates a second factor 
maximizing the variance explained in this second factor. The procedure continues until there are as many factors as 
variables. 



Rotated Factor Matrices 

Unrotated factor solutions achieve the objective of data reduction, but generally do not identify meaningful 
and adequate interpretation of the individual constructs and variables. The basic reason for employing a rotational 
method is to achieve simpler and theoretically more meaningful factor solutions. Rotation of the factors in most 
cases improves the interpretation by reducing some of the ambiguities which often accompany unrotated factor 
solutions. 

The 17 factors identified in Table 3 were subjected to Oblimin (obligue) rotation to improve the 
interpretation of data. The major option to the anaylyst in rotation is to choose between an orthogonal method or 
an oblique method. The oblique method of rotation was choosen for this study because the interest was to obtain 
theoretically meaningful constructs. In addition, an oblique factor rotation is more desirable because it is 
theoretically and empirically more realistic (Hair, Anderson, & Tatham, 1979). 
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Data interpretations from the Oblimin oblique rotated factor matrices were made from both the factor pattern 
matrix and the factor structure matrix. There is a basic difference between the pattern and structure matrices relevant 
for interpretation. The pattern loadings best show what variables are highly involved in what clusters. They 
distinctly display the patterns. The structure loadings however, do not display the patterns as well but rather 
measure the correlation of the variables with the factors (Rummel, 1970). Thompson (1997) argues that both factor 
pattern and structure coefficients should be interpreted in confirmatory factor analysis reports. Gorsuch (1983) also 
stated, “Indeed, proper interpretation of a set of factors can probably only occur if at least S [the factor structure 
coefficient matrix] and P [the factor pattern coefficient matrix] are both examined” (p. 208). 

A factor loading represents the correlation between the ADI descriptor (variable) and its respective factor. 
Loadings are zero (.00) when a variable is not related to a factor and are close to 1.0 when a variable is almost 
perfectly related to a factor. For the purposes of this study, the significance level for interpretation is .50 or greater. 

When subjecting the 17 extracted factors to a rotated factor analysis, the result was disappointing. Only six 
of the 13 ADI constructs loaded heavily on factors. Specifically, six of the constructs clustered in identifiable 
groups, as shown in Table 4. The other seven constructs had so few descriptors clustering in groups that 
identifying or labeling them was difficult. 

TABLE 4 

Results of the Rotated Factor Analyses 



Factor 1 



03 


Prob. Sol. 


.52 


16 


Prob. Sol. 


.84 


29 


Prob. Sol. 


.92 


42 


Prob. Sol. 


.79 


54 


Prob. Sol. 


.78 


67 


Prob. Sol. 


.82 


94 


Prob. Sol. 


.70 



Factor 12 

15 Organization .86 
28 Organization .70 
40 Organization .77 
53 Organization .51 
66 Organization .50 
93 Organization .59 



Factor 2 

04 Creativity .83 

1 7 Creativity . 70 

30 Creativity .77 

43 Creativity .62 

56 Creativity .77 

68 Creativity .57 

81 Creativity .55 

Factor 4 

41 Instr. Lead. .83 
48 Instr. Anal. .54 
55 Instr. Anal. .65 
61 Instr. Lead. .73 
69 Instr. Anal. .79 



Factor 8 

08 Communication .54 
21 Communication .62 
34 Communication .53 
47 Communication .73 
60 Communication .50 
73 Communication .55 
86 Communication .54 

Factor 6 

09 Instr. Lead. .58 
22 Instr. Lead. .58 
27 Instr. Anal. .74 
35 Instr. Lead. .74 
74 Instr. Lead. .80 
83 Instr. Anal. .62 
87 Instr. Lead. .73 
100 Instr. Lead. .60 
103 Instr. Anal. .84 
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The majority of significant descriptors (.50 or greater) loaded on 6 factors. Four factors noticeably identify 
discrete constructs as written for the ADI. Factor 1 consists of seven of the eight descriptors written for the ADI 
construct Problem Solving which implies a label or name for Factor 1 as Problem Solving. Factor 2 contains seven 
of the eight descriptors written for the construct of Creativity. Factor 8 contains seven of the eight descriptors written 
for the construct of communication, and Factor 12 contains six of the eight descriptors written for the construct of 
Organization. Factors 4 and 6 appear somewhat ambiguous, but upon careful analysis, both factors have heavy 
loadings of descriptors written for Instructional Analysis and Instructional Leadership. 

The constructs of climate development and instructional analysis revealed correlation matrices which 
consisted of lower correlation coefficients among the accompanying descriptors. In addition, two factors were 
identified in the rotated factor matrix of each of the constructs. In other words, the eight descriptors branched into 
separate areas, rather than measuring one discrete construct. 

DISCUSSION 

The use of truth tables by the ADI writing team resulted in the creation of 1 1 constructs with very strong 
items of measurement. For what reasons do the truth tables not have the same result with the constructs of climate 
development and instructional analysis? What causes these two constructs to split into two different factors? 

The belief is that the truth table did in fact, perform the appropriate function. The eight descriptors do 
measure a specified construct. The problem lies in the fact that the construct contains two factors. In a sense, the 
construct contains two constructs, with descriptors loading on one or the other. What might cause a descriptor to 

load on one factor rather than the other? Can the two constructs within one construct be identified? An attempt to 
answer these questions follows. 

The two factors of the construct climate development can be labeled as (a) the psycho-socio environment of 
management, and (b) the physical environment of management. Table 5 displays the rotated factor matrix for the 
eight descriptors of climate development. Two of the individual descriptors are in bold print to further clarify the 
ambiguity. 
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TABLE 5 

a t n x of A DI Constnict Cliin ate Peve lo p me^ t 



Descriptor 



Factor 1 Factor 2 



VOll 

V024 

V037 

VO50 

V063 

V076 

V089 

V102 



.36 

.67 

.11 

.87 

.02 

.96 

.84 

.39 



.61 

.09 

.84 

.08 

.72 

.05 

.06 

.65 



Descriptor V037 

attends to aspects of the physical environment to maximize learning. 
Descriptor V076 

maintains a facilitative psychological and social environment. 



Descriptor V076 loads highly on Factor 1 and implies attention to a psycho-socio detail. Descriptor 
VOj 7 on the other hand, clearly having a high factor loading on Factor 2, implies attention to a physical detail. 
Further inspection of the matrix shows that each descriptor loads heavily on either Factor 1 (psycho-socio 
environment) or Factor 2 (physical environment). The suggestion is not that the construct of climate development 
is an inappropriate construct of effective school administrators, but that the construct is comprised of two underlying 
constructs within one. Table 6 displays the wordings of the eight descriptors and the implied two factors of 
“psycho-socio environment” and “physical environment.” 

The difficulty seems to lie in the fact that the writing team inadvertently wrote four descriptors focused on 
the physical environment and four others focused on the psycho-socio environment. Two possible remedies exist: 
(a) rewrite the descriptors to load on one or the other, or (b) create two separate constructs, thus adding a construct 
to the ADI instrument. 

The same difficulty seems to exist with the ADI construct of instructional analysis which also loads on two 
factors. The two factors within the construct of instructional analysis are labeled as (a) emphasizing student 
achievement, and (b) instructional leadership emphasizing teaching, as shown in Table 7. This construct can be 
improved in the same manner as suggested above for the construct of climate development. 
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TABLE 6 

Climate Development Descriptors Loading of Two Factors 





PSYCHO-SOCIO ENVIRONMENT (FACTOR 1) 


Factor Loading 


Descriptor 24: works effectively with diverse elements 
of the school community. 


.67 


Descriptor 50: promotes a climate for psychological 
growth within the school environment. 


.87 


Descriptor 76: maintains a facilitative psychological 
and social environment. 


.96 


Descriptor 89: encourages and values contributions of 
all stakeholders. 


.84 


PHYSICAL ENVIRONMENT (FACTOR 2) 




Descriptor 1 1 : ensures a safe enviroment for learning. 


.58 


Descriptor 37: attends to the aspects of the physical 
environment to maximize learning. 


.84 


Descriptor 63: sets expectations for excellence within 
the school environment. 


.72 


Descriptor 102: positions the school as a community resource. 


.65 



TABLE 7 

Instructional Analysis Loading on Two Factors 



Descriptor 


Factor 1 


Factor 2 


V013 


.63 


.17 


V027 


.30 


.55 


V041 


.83 


.08 


V055 


.82 


.08 


V069 


.86 


.01 


V083 


.24 


.66 


V092 


.16 


.61 


VI03 


-.22 


.97 



Descriptor V041 

possesses high expectations for student achievement. 
Descriptor V083 

demonstrates expertise in analyzing teachers. 
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RECOMMENDATIONS 



A primary function of factor analysis is to reduce redundancy and ambiguity. The data analysis of the study 
clearly identifies the ADI constructs of climate development and instructional analysis as being indiscrete measures. 
These two dimensions are important skill dimensions of effective school leadership and should not be discarded as 
constructs. Again, the suggestion is not that these two constructs are inappropriate skill dimensions of effective 
school principals, but that the constructs as written contain two underlying constructs within one. Both constructs 
can be improved by rewriting descriptors to load on one rather than two factors. This could be accomplished by 
eliminating ambiguous and conflicting wording. 

Looking at the correlation matrix and communality table for each of the 13 ADI constructs, the data reveal 
1 or 2 descriptors with lower coefficients. Several of these descriptors also fail to load on any of the identified 17 
factors on the oblique rotated factor matrix. Repeating a factor analysis with a reduced number of descriptors 
(variables) is recommended. Eliminating the two lowest coefficients in each construct would reduce the variables 
from 104 to 78. The correlations between variables would likely improve and assist in further improving the 
construct validity of the ADI. 

The ADI writing team was careful to subject the eight descriptors within each construct to the truth table 
process. The team also utilized a 13 X 13 truth table matrix to test the 13 constructs for discreteness. The team 
failed however, to go further and create a 104 X 104 truth table matrix to test the eight descriptors within each 
construct with the decriptors of other constructs for discreteness. In other words, testing the construct definitions for 
disagreement on the truth table does not assure the descriptors within each construct to be discrete when compared 
to descriptors within other constructs. 

The truth table process which was so successful with the creation of descriptors within constructs can be of 
further help with the issue of construct correlations. The truth table process used for two-way agreement of 
descriptors can be equally valuable in seeking two-way disagreements. Though the constructs were worked through 
this process by the ADI writing team, the individual descriptors across constructs were not. 

Subjecting the descriptors to a 104 X 104 truth table is suggested for improving the discreteness of ADI 
constructs. Though the process would be difficult for one person, teams or groups could work through the process 
in a reasonable amount of time. If two-way disagreements could not be reached, individual descriptors would need 
to be reworded or modified in some way. Using a 104 X 104 truth table, descriptors within one construct can be 
compared with discriptors within another cosntruct. For example, a descriptor within the construct of problem 



solving compared with a descriptor in the construct of planning needs to result in a two-way disagreement. Though 
the correlation between decriptors within each construct should be high, the correlations across constructs should be 
low. 

An alternative recommendation for improving the discreteness of constructs involves the creation of a 
smaller and more manageable truth table. The data suggest six of the 13 constructs to be appropriately discrete. 

The creation of a truth table for the remaining seven would concentrate only on the ones appearing weak. This 
procedure would require a smaller truth table of 56 X 56. Even smaller 8X8 truth tables could be created to match 
existing high correlations from the factor matrix. For example, a high correlation exists between Factor 1 (problem 
solving) and Factor 2 (creativity). Using an 8 X 8 truth table matrix comparing the eight descriptors of problem 
solving with the eight descriptors of creativity would result in the tightening of construct discreteness. 



The revised Administrative Diagnostic Inventory (ADI) shows significant improvements in content validity 
and more specifically, construct validity of the instrument. Factor analysis reveals the descriptors within each of the 
13 skill dimensions (constructs) to be highly correlated and significant measures of the construct for which they were 
written. 

Factor analysis also reveals six of the 13 constructs to be discrete and separate measures of effective behavior 
of school principals. The seven constructs which did not prove discrete can likely be improved by subjecting the 
104 descriptors to the verbal logic of truth tables. 

Even though the ADI contains some weaknesses, the instrument proves to be significantly improved over 
the previous model. Including so many practitioners from the field and professionals from several universities in the 
revision process can be credited with the successful improvement of the ADI. 

The problem of reliability in factor analysis is always an issue. Like any other statistical procedure, a factor 
analysis starts with a set of imperfect data. When the data change because of changes in the sample, the data 
gathering process, or the many kinds of measurement errors, the results of the analysis will change also. Therefore, 
the results of any single analysis are not entirely dependable. If the result of this study has been to spur further 
research and analysis of the Administrative Diagnostic Inventory, an important purpose is realized. As the 
leadership construct constantly evolves, regular and ongoing validity studies are crucial and necessary. 



SUMMARY 




19 



22 



References 



Adams, R.J. (1996). Confirmatory factor analysis. Unpublished manuscript, 
California State University Fresno. 

Brubaker, L. (1982). The assessment center method: Developing employees of 
mid-mangement positions. Journal of College and University Personnel Association, 34, 
10-36. 

Coleman, D.G. (1973). Improving confirmatory factor loadings through use of 
symbolic-logic matrices. Unpublished manuscript, Kirksville, MO: Northeast Missouri 
State University. 

Coleman, D.G. (1995). Construct validity: Administrative Diagnostic Inventory 
for the National Association of Elementary School Principals. Unpublished manuscript, 
California State University Fresno. 

Coleman, D.G. and Creighton, T.B. (1996). Testing confirmatory factoring 
procedures. Paper presented at the National Council of Professors of Educational 
Administration 50th Anniversity Conference, Corpus Christi, TX. 

Comrey, A.L. (1973). A first course in factor analysis. Los Angeles, CA: 
Academic Press. 

Creighton, T.B. (1996). A construct validation of the Administrative Diagnostic 
Inventory. Unpublished doctoral dissertation, University of California Davis/California 



State University Fresno. 



Durden, P.C. (1994). A predictive validity study of an administrator assessment 



system for principal selection and professional development. Unpublished doctoral 
dissertation. University of Washington. 

Edmonds, R. (1982). Program of school improvement: A 1982 overview (ERIC 
No. ED 221 536). Arlie House, VA: The National Institute of Education. 

Elsaesser, L.A. (1990). The relationship between assessment center participation 
and subsequent staff development activities in practicing and aspiring principals. 
Unpublished doctoral dissertation, University of Washington. 

Glassman, N. and Heck, R. (1992). The changing role of the principal: 
Implications for principal assessment. Peabody Journal of Education, 68 (3), 2-30. 

Gorsuch, R.L. (1983). Factor analysis. Hillsdale, NJ: Lawrence Erlbaum. 

Guilford, J.P. (1946). New standards for test evaluation. Educational and 
Psychological Measurement, 6, 427-439. 

Hair, J.F. , Anderson, R.E., and Tathum, R.L. (1979). Multivariate data analysis 
with readings. Tulsa: Petroleum Publishing Company. 

Hallinger, P. (1992). Changing norms of principal leadership in the United States. 
Journal of Educational Administration. 30 (3), 35-48. 

Leithwood, K. and Hallinger, P. (1993). Cognitive perspectives on educational 
administration: An introduction. Educational Administration Quarterly, 24 (3), 296-301. 

Lipman, J.A. (1981). Effective principal, effective school. NASSP Bulletin, 48 
(2), 67-84. 




24 



Nunnally, J.C. (1978). Psychometric theory. New York: McGraw-Hill. 

Sirotnik, K.A. and Durden, P C. (1996). The validity of administrator 
performance assessment systems: The ADI as a case in point. Educational 
Administration Quarterly, 32 (4), 539-564. 

Smith, W.F. and Andrews, R.L. (1989). Instructional leadership: How principals 
make a difference. Alexandria, VA: Association for Supervision and Curriculum 
Development. 

Thompson, B. (1997). The importance of structure coefficients in structural 
equation modeling confirmatory factor analysis. Educational and Psychological 



Measurement, 57 (1), 5-19. 



ERIC FACILITY 



ID : 301-953-0263 



APR 14 ’9? 



15=14 No .007 P.02 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
Educational Resources Information Center (ERIC) 

REPRODUCTION RELEASE 

(Specific Document) 



» 




7 M 07-T D ^- L i 



I. DOCUMENT IDENTIFICATION: 




II. REPRODUCTION RELEASE: 



In order to d&camlnate as widely as possible timely and significant materials of Interest to the educational community, documents announced 
In the monthly abstract Journal of the ERIC system, Resources In Education (RIE), are usually made available to users in microfiche, reproduced 
paper copy, and electronic/optical media, and sold through the ERIC Document Reproduction Servioe (EDRS) or other ERIC vendors. Credit is 
given to the source of each document, and, If reproduction release Is granted, one of the following notices is affixed to the document. 



If permission Is granted to reproduce and disseminate the identified document, please CHECK ONE of the following two options and sign at 
the bottom of the page. 



The sample sticker shown below will be 

ET 

t 

Check here 
For Level 1 Release: 

Permitting reproduction in 
microfiche (4* x 6* film) or 
other ERIC archival media 
(e.g., electronic or optical) 
and paper copy. 



The sample sticker shown below will be 
affixed to all level 2 documents 

□ 

♦ 

Check here 
For Level 2 Release: 

Permitting reproduction in 
microfiche (4* x 6" film) or 
olher ERIC archival media 
(e.g., electronic or optical), 
but not In paper oopy. 



affixed to all Level 1 documents 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL 
HAS BEEN GRANTED BY 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS 
MATERIAL IN OTHER THAN PAPER 
COPY HAS BEEN GRANTED BY 






$ 



& 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



Level 1 



Level 2 



Documents will bB processed as Indicated provided reproduction quality permits. If permission 
to reproduce Is granted, but neither box Is checked, documents will be processed at Level 1. 



Sign 

here-* 

please 



V hereby grant to the Educational Resource s Information Center (ERIC) nonexclusive permission to reproduce and disseminata 
this document as Indicated above. Reproduction from the ERIC microfiche or electronic/optical madia by persons other than 
ERIC employees and its system contractors requires permission from the copyright holder. Exception Is made for non-profit 
reproduction by libraries and other service agencies to satisfy Information needs of educators In response to discrete inquiries. m 




Printed Name/P osrtion/TItJ e: 


OrganizatiorvAddress: A 

(/Odt&rs/f’y o r 

/2.£" Coff-e^p* T>J,P-e 
OiSfzr, tVYQAiiAj 0/ 


Telephone; ^ 

3of pGt-zsry 


Tax* 


Cr4tjf»fa & 4C4*.cc.. 
cr> . i&dts 


‘Date” 



(over) 




